Remote presence system including a cart that supports a robot face and an overhead camera

ABSTRACT

A tele-presence system that includes a cart. The cart includes a robot face that has a robot monitor, a robot camera, a robot speaker, a robot microphone, and an overhead camera. The system also includes a remote station that is coupled to the robot face and the overhead camera. The remote station includes a station monitor, a station camera, a station speaker and a station microphone. The remote station can display video images captured by the robot camera and/or overhead camera. By way of example, the cart can be used in an operating room, wherein the overhead camera can be placed in a sterile field and the robot face can be used in a non-sterile field. The user at the remote station can conduct a teleconference through the robot face and also obtain a view of a medical procedure through the overhead camera.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The subject matter disclosed generally relates to the field of robotictele-presence systems.

2. Background Information

Robots have been used in a variety of applications ranging from remotecontrol of hazardous material to assisting in the performance ofsurgery. For example, U.S. Pat. No. 5,762,458 issued to Wang et al.discloses a system that allows a surgeon to perform minimally invasivemedical procedures through the use of robotically controlledinstruments. One of the robotic arms in the Wang system moves anendoscope that has a camera. The camera allows a surgeon to view asurgical area of a patient.

There has been marketed a mobile tele-presence robot introduced byInTouch Technologies, Inc., the assignee of this application, under thetrademark RP-7. The InTouch robot is controlled by a user at a remotestation. The remote station may be a personal computer with a joystickthat allows the user to remotely control the movement of the robot. Boththe robot and remote station have cameras, monitors, speakers andmicrophones to allow for two-way video/audio communication. The robotcamera provides video images to a screen at the remote station so thatthe user can view the robot's surroundings and move the robotaccordingly.

InTouch also provides a system sold as VisitOR that includes a robotface that is attached to a boom. The boom and robot face can beinstalled into an operating room. Using a robot face in an operatingroom may require sterilization of the face. Additionally, the VisitORrequires the installation of a boom in the operating room. This can addto the cost and complexity of installing such a system.

BRIEF SUMMARY OF THE INVENTION

A tele-presence system that includes a cart. The cart includes a robotface that has a robot monitor, a robot camera, a robot speaker, a robotmicrophone, and an overhead camera. The system also includes a remotestation that is coupled to the robot face and the overhead camera. Theremote station includes a station monitor, a station camera, a stationspeaker and a station microphone.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a tele-presence system;

FIG. 2 is a perspective view of a cart of the system;

FIG. 3 is a rear view of an articulated arm and a robot face of thecart;

FIG. 4 is an enlarged perspective view of an overhead camera of thecart;

FIG. 5 is an illustration of a display user interface of a remotestation;

FIG. 6 is an illustration of the display user interface showing videoimages captured by a robot camera and an overhead camera beingsimultaneously displayed.

DETAILED DESCRIPTION

Disclosed is a tele-presence system that includes a cart. The cartincludes a robot face that has a robot monitor, a robot camera, a robotspeaker, a robot microphone, and an overhead camera. The system alsoincludes a remote station that is coupled to the robot face and theoverhead camera. The remote station includes a station monitor, astation camera, a station speaker and a station microphone. The remotestation can display video images captured by the robot camera and/oroverhead camera. By way of example, the cart can be used in an operatingroom, wherein the overhead camera can be placed above a sterile field toprovide a more advantageous vantage point to view a procedure. The userat the remote station can conduct a teleconference through the robotface and also obtain a view of a medical procedure through the overheadcamera.

Referring to the drawings more particularly by reference numbers, FIGS.1, 2 and 3 show a tele-presence system 10. The system 10 includes a cart12 that is coupled to a remote control station 14. The cart 12 has arobot face 16 and an overhead camera 18. The remote control station 14may be coupled to the cart 12 through a network 20. By way of example,the network 20 may be either a packet switched network such as theInternet, or a circuit switched network such as a Public SwitchedTelephone Network (PSTN) or other broadband system. Alternatively, thecart 12 may be coupled to the remote station 14 network thru asatellite.

The remote control station 14 may include a computer 22 that has amonitor 24, a camera 26, a microphone 28 and a speaker 30. The computer22 may also contain an input device 32 such as a joystick or a mouse.The control station 14 is typically located in a place that is remotefrom the cart 12. Although only one remote control station 14 is shown,the system 10 may include a plurality of remote stations 14. In generalany number of carts 12 may be coupled to any number of remote stations14 or other carts 12. For example, one remote station 14 may be coupledto a plurality of carts 12, or one cart 12 may be coupled to a pluralityof remote stations 14, or a plurality of carts 12. The system mayinclude an arbitrator (not shown) that control access between the carts12 and the remote stations 14.

As shown in FIG. 3, the cart 12 may include an articulated arm 40 thatsupports and can move the robot face 16. The articulated arm 40 may haveactive joints 42 and 44 that allow the robot face 14 to be panned andtilted, respectively. The active joints 42 and 44 may move in responseto commands provided by the remote station. The joints 42 and 44 maycontain position sensors 46 and 48, respectively, that providepositional feedback of the arm 40.

Referring to FIGS. 2 and 3, each robot face 16 includes a camera(s) 50,a monitor 52, a microphone(s) 54 and a speaker(s) 56. The robot camera50 is coupled to the remote monitor 24 so that a user at the remotestation 14 can view a video image captured by the robot camera 50.Likewise, the robot monitor 52 is coupled to the remote camera 26 sopersonnel at the surgical site may view the user of the remote station14. The microphones 28 and 54, and speakers 30 and 56, allow for audiblecommunication between the system operator and the personnel at thesurgical site.

The overhead camera 18 may be coupled to a boom 60. The boom 60 mayinclude a number of joints 62, either active or passive. The joints 62may include positional sensors to provide feedback regarding theposition of the overhead camera 18.

As shown in FIG. 4, the cart 12 may include an overhead microphone 64and a detachable handle 66. The overhead microphone 64 may provide analternative source of sound. The detachable handle 66 can be used tomove the boom 60 and overhead camera 18. If the cart 12 is used in asterile field, for example in an operating room, the handle 66 may bereplaced with a sterile handle before each medical procedure to allow asurgeon within the sterile field to position the boom during aprocedure.

Referring again to FIG. 2, the cart 12 may include a linear actuator 70that can be remotely or locally actuated to vary the height of the robotface 16 and overhead camera 18. Varying the height allows the cart 12 tobe rolled through doors and then actuated to move the face 16 and camera18 to elevated positions. For example, the face 16 and camera 18 can belowered to allow the cart 12 to be moved into an operating room. Thecamera 18 can then be raised to provide a desirable view over anoperating table. The cart 12 may include a laser pointer 72 and/ordirected lighting (not shown) located on the boom 60. The cart 12 mayalso include a local control panel 74 to move the articulated arm 40,actuator 70 and/or boom 60. The linear actuator 70 is also advantageousin moving the face 16 to be essentially at the same level as a personwhether they are standing, sitting or lying in a prone position.

The robot face 16 may include a processor, hard disk drive and othercircuits that enable the face 16 to function as a computer. The face 16may include an input panel 76 that allows a user to provide input. Byway of example, the operator of the remote station may provide one ormore questions through the robot face 16, wherein a user of the cartprovides answers through the input panel 76.

The system 10 may have certain components and software that are the sameor similar to a robotic system provided by the assignee InTouchTechnologies, Inc. of Goleta, Calif. under the name RP-7 and embodies asystem described in U.S. Pat. No. 6,925,357, which is herebyincorporated by reference.

FIG. 5 shows a display user interface (“DUI”) 120 that can be displayedat the remote station 14. The DUI 120 may include a robot view field 122that displays a video image captured by the robot camera and/or theoverhead camera. The DUI 120 may also include a station view field 124that displays a video image provided by the camera of the remote station14. The DUI 120 may be part of an application program stored andoperated by the computer 22 of the remote station 14.

The DUI 120 may include a graphical switch 126 that allows the user toselect between the video image provided by the robot camera and thevideo image provided by the overhead camera. The DUI 120 may also have agraphical switch 128 that allows the user to select the simultaneousdisplay of the video images from the robot and overhead cameras as shownin FIG. 6. The video images from both cameras can be streamed to theremote station from the cart. The images can be merged by presenting acenter rectangle of each image (e.g., 320×480 center area). A zoom orhighlighting feature may be utilized by manipulating a cursor on eitherimage. The system may also automatically pan a camera when the cursor ismoved out of the displayed field of view.

The system may automatically present the video image from a camera thathas an optimal view of an object. For example, the system may utilizepattern recognition techniques to determine which video image provides amore clear image of an object. The system may determine which camera isin closer proximity to an object and provide the image from the camerathat is closer to the object. The system may utilize positional feedbackfrom the cart to determine the proximity of the cameras relative to theobject. The system may also have sensors, such as laser, sonar, etc.that can determine the proximity of the cameras to the object. Thesystem may use the feedback and/or sensors to determine which camera iscloser to an object.

The system may automatically move the cameras so that each camera ispointed to the same or substantially the same field of view. Forexample, if the robot face is pointed toward an object, the overheadcamera can be automatically moved to capture a video image of the sameobject. Likewise, if the overhead camera is capturing a video image ofan object, the robot face can be automatically moved to point toward thesame object. This enhances the “presence” of the remote operator becausethey are facing the same object that the overhead camera is viewing.

The DUI 120 may have a graphical switch 130 that allows the user toswitch between sound captured by the robot microphone or the overheadmicrophone. The system may automatically switch between microphonesbased on a characteristic(s) of the sound captured by the microphones.For example, the system may switch to the microphone that provides thehighest aural clarity, or to the microphone that is in the closestproximity to a person or object generating the sound.

The DUI 120 may include a location display 138 that provides thelocation of the robot face. The CHANGE button 140 can be selected tochange the default robot face in a new session. The CHANGE button 140can be used to select and control a different robot face in a systemthat has multiple robot faces. The user can initiate and terminate asession by selecting box 142. The box 142 changes from CONNECT toDISCONNECT when the user selects the box to initiate a session. Systemsettings and support can be selected through buttons 144 and 146.

Both the robot view field 122 and the station view field 124 may haveassociated graphics to vary the video and audio displays. Each field mayhave an associated graphical audio slide bar 148 to vary the audio levelof a selected microphone and another slide bar 152 to vary the volume ofthe speakers.

The DUI 120 may have slide bars 150, 154 and 156 to vary the zoom, focusand brightness of a selected camera, respectively. A still picture maybe taken at either the robot face or remote station by selecting one ofthe graphical camera icons 158. The still picture may be the imagepresented at the corresponding field 122 or 124 at the time the cameraicon 158 is selected. Capturing and playing back video can be takenthrough graphical icons 160. A return to real time video can be resumed,after the taking of a still picture, captured video, or reviewing aslide show, by selecting a graphical LIVE button 162.

A still picture can be loaded from disk for viewing through selection oficon 164. Stored still images can be reviewed by selecting buttons 166.The number of the image displayed relative to the total number of imagesis shown by graphical boxes 168. The user can rapidly move through thestill images in a slide show fashion or move through a captured videoclip by moving the slide bar 170. A captured video image can be pausedthrough the selection of circle 174. Play can be resumed through thesame button 174. Video or still images may be dismissed from the activelist through button 172. Video or still images may be transferred to therobot by selecting icon 176. For example, a doctor at the remote stationmay transfer an x-ray to the screen of the robot.

The system may provide the ability to annotate 184 the image displayedin field 122 and/or 124. For example, a doctor at the remote station mayannotate some portion of the image captured by the robot face camera.The annotated image may be stored by the system. The system may alsoallow for annotation of images sent to the robot face through icon 176.For example, a doctor may send an x-ray to the robot face which isdisplayed by the robot screen. The doctor can annotate the x-ray topoint out a portion of the x-ray to personnel located at the robot site.This can assist in allowing the doctor to instruct personnel at therobot site.

The display user interface may include graphical inputs 186 that allowthe operator to turn the views of the remote station and remote camerason and off.

Referring to FIG. 1, the cart 12 can be used in an operating room. Byway of example, the boom 60 can be moved to place the overhead camera 18above an operating table 200. The overhead camera 18 may be locatedabove a sterile field. The robot face 16 may be placed adjacent to thesterile field. With such a configuration, personnel may conduct two-wayvideo conferencing through the robot face 16. The overhead camera 18 mayprovide a more desirable view of the patient and operating procedure.This would allow a physician at the remote station to view the procedureand have a video conference to provide instructions, mentoring, etc. topersonnel at the surgical site.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art.

What is claimed is:
 1. A surgical mentoring system comprising: a cartincluding: a first camera; a second camera; a speaker; a microphone; andan input device, wherein the second camera is coupled to the cart via aboom that includes one or more joints configured to allow a distancebetween a first end of the boom and a second end of the boom to bevaried, the cart is configured to be coupled to a remote station via anetwork, the remote station including a camera, a monitor, a microphone,and a speaker, wherein the remote station is configured to display auser interface on the remote station monitor, the user interfaceincludes a remote view field, a remote station view field, and aplurality of graphical controls, the plurality of graphical controlsincludes two or more of a control to selectively display video from thefirst camera or the second camera, a control to display a still imagecaptured by the first camera or the second camera in the remote viewfield, a control to selectively display an image captured by the remotestation camera on the cart monitor, and a control allowing a user of theremote station to annotate an image displayed on the cart monitor fromthe remote station, the station view field displays an image captured bythe station camera, the station speaker reproduces a sound captured bythe cart microphone, the station microphone captures a sound that istransmitted to and reproduced by the cart speaker, and the orientationof the first camera and the orientation of the second camera can beindependently controlled via an input device of the remote station. 2.The system of claim 1, wherein the two or more controls includes aswitch that allows the user to select simultaneous display of videoimages from the first and second cameras.
 3. The system of claim 1,wherein the video images from both the first camera and the secondcamera are streamed to the remote station from the cart.
 4. The systemof claim 1, wherein a height of the first camera is less than a heightof the second camera.
 5. The system of claim 1, wherein the boomincludes a handle that can be used to move the second camera.
 6. Thesystem of claim 1, wherein the cart includes a processor and a harddrive.
 7. The system of claim 1, wherein cart includes an input panelthat allows a user to provide input.
 8. The system of claim 7, whereinthe user interface includes a location display that provides thelocation of the cart.
 9. The system of claim 7, wherein the systemincludes a plurality of carts and the user interface includes agraphical input that can be used to select and control a second cart.10. The system of claim 1, wherein said cart includes a secondmicrophone.
 11. The system of claim 1, wherein the one or more controlsincludes a control that allows a user to switch between the firstmicrophone and the second microphone.
 12. A method for surgicalmentoring, comprising: displaying a user interface on a monitor of aremote station, the user interface including a remote view field, astation view field, and a plurality of graphical controls; establishinga communication session with a cart in proximity to a surgical field,the cart including a first camera, a second camera, a monitor, amicrophone, a speaker, and an input device, wherein the second camera iscoupled to the cart via a boom that includes one or more jointsconfigured to allow a distance between a first end of the boom and asecond end of the boom to be varied; displaying, in the station viewfield, an image captured by the station camera on the station monitor,reproducing, by the station speaker, a sound captured by the cartmicrophone; capturing, by the station microphone, a sound that istransmitted to and reproduced by the cart speaker; and controlling theorientation of the first camera and the orientation of the second camerafrom the remote station, wherein the plurality of graphical controlsincludes controls for two or more of selectively displaying video fromthe first camera or the second camera in the remote view field inresponse to user input at the remote station, displaying a still imagecaptured by one of the first camera or the second camera in the remoteview field in response to user input at the remote station, selectivelydisplay an image captured by the remote station camera on the cartmonitor, and annotating an image displayed on the cart monitor inresponse to user input at the remote station.
 13. The method of claim13, further comprising simultaneously displaying video images from thefirst and second cameras at the remote station.
 14. The method of claim13, further comprising receiving at the remote station video streamsfrom both the first and second cameras.
 15. The method of claim 13,further comprising receiving input from an input panel of the cart atthe remote station.
 16. The method of claim 13, further comprisingdisplaying a location display at the remote station that provides thelocation of the cart.
 17. The method of claim 13, further comprisingdisplaying at the remote station a graphical input that can be selectedto select and control a second cart.
 18. The method of claim 13, furthercomprising controlling the zoom of the first camera from the remotestation in response to user input.
 19. The method of claim 13, furthercomprising controlling the zoom of the second camera from the remotestation in response to user input.
 20. A non-transitorycomputer-readable medium comprising program code that, when executed byone or more processors, causes the one or more processors to perform amethod for automated health condition scoring, the method comprising:displaying a user interface on a monitor of a remote station, the userinterface including a remote view field, a station view field, and aplurality of graphical controls; establishing a communication sessionwith a cart in proximity to a surgical field, the cart including a firstcamera, a second camera, a monitor, a microphone, a speaker, and aninput device, wherein the second camera is coupled to the cart via aboom that includes one or more joints configured to allow a distancebetween a first end of the boom and a second end of the boom to bevaried; displaying, in the station view field, an image captured by thestation camera on the station monitor, reproducing, by the stationspeaker, a sound captured by the cart microphone; capturing, by thestation microphone, a sound that is transmitted to and reproduced by thecart speaker; and controlling the orientation of the first camera andthe orientation of the second camera from the remote station, whereinthe plurality of graphical controls includes controls for two or more ofselectively displaying video from the first camera or the second camerain the remote view field in response to user input at the remotestation, displaying a still image captured by one of the first camera orthe second camera in the remote view field in response to user input atthe remote station, selectively display an image captured by the remotestation camera on the cart monitor, and annotating an image displayed onthe cart monitor in response to user input at the remote station.